Operating systems typically employ monolithic kernels that provide basic services for all other parts of the operating system. These kernels form part of a trusted computing base (TCB) of the operating system (OS), which is the set of all hardware, firmware, and/or software components that are critical to the security of the operating system, in the sense that bugs or vulnerabilities occurring inside the TCB might jeopardize the security properties of the entire system. As the code that makes up the kernel is needed continuously, it is usually loaded into computer storage in an area that is protected. Attacks by malware and rootkits can add code or replace portions of the core operating system at both the kernel and its associated device drivers. Kernel-mode attacks against the trusted computing base can be difficult to detect since they operate at the same security level as the operating system.
In response to kernel-mode attacks against large TCB operating systems, security hypervisors are used. In some implementations, a small TCB security hypervisor with hardware virtualization is used to protect the OS's memory and central processing unit (CPU) state changes. Hardware virtualization can include, for example, Extended Page Tables (EPTs) in conjunction with a trusted in-guest agent running inside the operating system. These systems may detect memory events such as, for example, external interrupts or attempts to access privileged registers or resources. However, detecting an event may cause a virtual machine exit (VM exit) when an event, operation or situation, such as an external interrupt, attempts to access privileged registers or resources. This event, operation or situation is intercepted causing the virtual processor to exit the virtual machine environment through the VM exit so that the hypervisor may implement virtualization policies.
Other existing solutions require a significant amount of code and data to be resident in the same operating system that the solutions are trying to protect. If there is any flaw in either the policy or the code that implements the protection, malware could potentially completely disable the solution. The large amount of code and data not only represents a large attack surface, it also represents a large detection surface, which can be a potentially significant issue for products that use virtualized sandboxes for behavioral analysis and are susceptible to evasion by sophisticated malware.
Other solutions include using general purpose hypervisors for monitoring OS memory and CPU state. In these solutions, the operating system is running in a first guest VM that is to be protected with protection software running in a second, privileged, guest VM. However, this approach has a very high overhead through multiple VM exits/entries as enforcing policies may induce many memory access events that need the first guest virtual machine to be de-scheduled and the second guest virtual machine to be scheduled synchronously. Also, CPU assists that can avoid VM exits such as virtualization exceptions cannot operate across CPU threads currently and hence cannot be leveraged across guest domains either.
A technique for protecting/monitoring an operating system without the high-overhead of VM exits while also providing the benefits of low-latency and the flexibility of an agent operating in a separate VM would be desirable.